Python Start Blog

Regular expressions in Python on Raspberry Pi

info@pythonstart.com (Python Start) — Sun, 28 Jun 2020 04:05:53 GMT

Regular expressions are a powerful tool to parse and validate text input

Python includes many built-in functions to work with strings, but as you start working with increasing volumes of text input, you need a more powerful and flexible way to validate and format text. As an example, below is a screenshot of a simple Python program running on a Raspberry Pi. This program opens an HTML file called Good morning.html , stores the contents of the file in memory as a string and then prints the string to the console.

The file contents look disorganized and a bit overwhelming when viewed as a string, but if you download the HTML file and then open it in a web browser, you will discover that it is a very simple, interactive story.

Inside the HTML file, there is a lot of text. If you look closely, you will notice the story text is prefaced with an HTML tag, <tw-storydata> . In this example, we are only interested in the story text, so we decide to extract certain portions of the file using regular expressions.

To do this, we start by importing the regular expression module with the import re statement. Next, we read in the file contents and store it in a string. Then we search the file contents for the <tw-storydata> tag using the re.search() method. Take a look at the code below, then let's dive into the details of the regular expression.

'''Regular expressions in Python'''

import re

# Code to open the local file if you have downloaded it

with open( "Good morning.html" ) as f:

file_text = f.read()

story_data = re.search( '\

<tw-storydata.*?\

name="(?P<story_name>.*?)"\

.*</tw-storydata>' , file_text)

print( "Story name: '" + story_data.group( "story_name" ) + "'" )

While the regular expression above may look complicated, the goal is simple: we want to find and store the story name. Let's look at it step by step to see how it works:

We start by searching for all of the story data. This is the text between the <tw-storydata> opening and closing HTML element
To do this, we pass the re.search() method a string parameter that contains the regular expression to describe what we are seeking
Start with a backslash character ( \ ) to wrap to the next line to make the regular expression easier to read
Add the <tw-storydata opening element that we are looking for
The period character ( . ) indicates that we are looking for any character
The asterisk character ( * ) modifies the search to look for any number of characters
The question mark ( ? ) modifies the search to be non-greedy, so the search will stop at the first instance of name=" encountered after the <tw-storydata opening element
Another period + asterisk + question mark + closing angled bracket ( .*?> ) indicates a non-greedy search for a closing bracket
The next step is to look for and capture the story name, so we search for name=" attribute then use the Python syntax for a named group syntax (?P<story_name>.*?)
We round out the search with the closing element to validate that the structure conforms to our expectations.

Once the regular expression has run, we print out the story name by looking up the named group, which we called " story_name" .

for passage in re.findall( '\

<tw-passagedata.*?\

name="(.*?)"\

.*?>\

(.*?)\

</tw-passagedata>' , file_text):

print(passage[0] + ": '" + passage[1] + "'" )

The snippet of code finds and captures each passage within the story. Here's how it works:

We loop through all the passages that are found match the pattern we define
The <tw-passagedata> elements have a variety of attributes, but we only capture text within name="" by using capturing groups with the (.*?)
We also capture the actual passage by doing a non-greedy search for the closing angle bracket .*?> then use another capturing group
We then print the passage name and the actual passage text.

Ready to give it a try? Jump in and play with this example , complete with the code needed to import the HTML file from Google Drive.

Follow along with this video to help you see how to run this on a Raspberry Pi. Let us know if you have any questions by leaving a comment below.

Solving linear programming problems in Python with PuLP

Mon, 22 Jun 2020 06:29:11 GMT

See how to solve a staffing problem with PuLP, a linear programming toolkit for Python

Linear programming is a way to find ideal solutions to linear functions with multiple variables. For example, reducing project timelines by minimizing critical paths or maximizing revenue with an optimal product mix.

To better understand the benefits of linear programming in real world scenarios, consider the 2014 paper by Mr. B.Satheesh Kumar, Ms. G. Nagalakshmi and Dr. S. Kumaraguruabout . The authors studied and optimized a hospital's scheduling process to maximize fairness for individual nurses while satisfying a variety of hospital requirements. An algorithm was used, consisting of a set of linear functions and variables to describe the staffing parameters, and then linear programming was used to minimize the number of staff required while meeting hospital requirements and maximizing fairness among the individual nurses.

Creating optimal schedules for hospital nursing staff is complex. Schedules must balance work hours among the nurses while adhering to strict healthcare requirements. For example, nurses may be subject to a maximum number of consecutive workdays and a minimum amount of rest time between shifts. In addition, hospitals typically need to minimize costs, while ensuring that they have sufficient staff for each time period.

Nurses report at the hospital at the beginning of each period and work for 8 consecutive hours. The hospital wants to determine the minimal number of nurses to be employed so that there will be a sufficient number of nurses available for each period.

To demonstrate how linear programming can help, we start by solving an example in Excel. Blue cells represent inputs, yellow cells represent variables and the orange cell represents our objective function. In this case, we want to minimize the total number of nurses required, while meeting certain minimum staffing levels for each 4-hour time window. The nurse staffing schedule is represented in an Excel table, with 4-hour time windows as rows, 8-hour shifts as columns and a blue matrix of 1s and 0s to show how shifts overlap.

To make this model easier to manage, four named ranges are described in the gray table on the far right:

The minimum number of nurses for each 4-hour time window are listed in the right-most blue column
The number of nurses needed for each 8-hour shift is shown in the row of yellow cells
Definitions of which shifts work each 4-hour time window are defined in the blue cells at C4:H9
The total number of nurses working during any given 4-hour time window are shown in column I

These cell ranges are defined and can be viewed in Excel's Name Manager from the "Formulas" Ribbon tab.

This sheet automatically optimizes the schedule using Solver, an Excel add-in. Solver automatically adjusts the default values in the yellow cells to optimize staffing for each 8-hour shift, ensuring adequate staff for each 4-hour window and minimizing the total number of nurses required.

Before we run Solver, notice that our objective is to minimize the total number of nurses required, subject to two types of constraints:

The number of nurses must be an integer (whole number) to avoid fractional staffing numbers
The number of nurses working in any 4-hour window must be greater than or equal to the minimum

Once all of the Solver parameters are set, Solver is run to find an optimal solution.

After just a moment, Solver displays a message that indicates that an optimal solution has been found that meets our constraints. Click "OK" and check to be sure the results are accurate.

The resulting schedule meets the staffing constraints for each 4-hour window, while minimizing the total number of nurses required. Now let's take a look at how this can be done in Python using PuLP . The first step is to install PuLP using the command pip install pulp .

Here is some sample code, written in Python, that uses PuLP to solve this linear programming problem. We start by importing the pulp library, then setup a pulp.LpProblem object with a string description and the pulp.LpMinimize parameter to indicate that we want to minimize the number of staff. Next, we create variables to represent the number of nurses working each shift, then define the objective function by adding up the number of nurses for each shift. Finally, we add the six constraints to represent the minimum number of nurses for each 4-hour time window. For example, we need a minimum of 70 nurses for the first 4-hour time window from 6 - 10 a.m.

''' Solving a staffing linear programming problem with PuLP in Python '''

import pulp

prob = pulp.LpProblem( "Nurse Staffing" , pulp.LpMinimize)

# Variables to represent the number of nurses working each shift

shift1=pulp.LpVariable( "Shift1" , 0 , None ,pulp.LpInteger)

shift2=pulp.LpVariable( "Shift2" , 0 , None ,pulp.LpInteger)

shift3=pulp.LpVariable( "Shift3" , 0 , None ,pulp.LpInteger)

shift4=pulp.LpVariable( "Shift4" , 0 , None ,pulp.LpInteger)

shift5=pulp.LpVariable( "Shift5" , 0 , None ,pulp.LpInteger)

shift6=pulp.LpVariable( "Shift6" , 0 , None ,pulp.LpInteger)

# The objective function is added to 'prob' first

prob += shift1 + shift2 + shift3 + shift4 + shift5 + shift6, "Total Nurses"

# The six constraints to ensure there are enough nurses for each time period

prob += shift6 + shift1 >= 70 , "MimimumNurses06amTo10am"

prob += shift1 + shift2 >= 170 , "MimimumNurses10amTo2pm"

prob += shift2 + shift3 >= 200 , "MimimumNurses02pmTo6pm"

prob += shift3 + shift4 >= 85 , "MimimumNurses06pmTo10pm"

prob += shift4 + shift5 >= 25 , "MimimumNurses10pmTo2am"

prob += shift5 + shift6 >= 40 , "MimimumNurses02amTo06am"

# Solve the problem using PuLP's choice of Solver

prob.solve()

# The status of the solution is printed to the screen

print( "Status:" , pulp.LpStatus[prob.status])

# Each of the variables is printed with it's resolved optimum value

for v in prob.variables():

print(v.name, "=" , v.varValue)

# The optimized objective function value is printed to the screen

print( "Total number of nurses = " , pulp.value(prob.objective))

Now, we are ready to solve using the LpProblem object's solve() method, then print out the results of each variable.

Notice in the output show in figure 7 below that the number of nurses for each shift are different from those we found using Solver in Excel, but each of the constraints is met and the total, minimum number of nurses is the same: 295.

Now we can validate the PuLP results by plugging the shift numbers into our Excel model as shown in Figure 8 below. Even though the 10 p.m. - 6 a.m. shift has 0 nurses, the 25 nurses who started during the previous shift from 6 p.m. - 2 a.m. are sufficient to ensure there are 25 nurses available during from 10 p.m. - 2 a.m.

In this Python example, we have seen how linear programming can help solve real world problems and how these can be solved in Python using PuLP. If you would like to try running this code sample, access it here .

If you have any questions, please leave a comment below or contact us. We look forward to seeing how you use linear programming in Python to solve your real world challenges.

The list of Python Packages is Massive

info@pythonstart.com (Python Start) — Sat, 04 Mar 2017 08:00:00 GMT

Python's geometric rise is due, at least in part, to a tremendous list of packages

Python has experienced tremendous recent adoption for at least 5 reasons . One of those reasons is the vast set of available Python packages.

What is a Python Module?

In order to understand Python packages , it's helpful to first understand Python modules . Modules are individual Python files that can be imported and interpreted at run-time. For example, you could create a file named log_search.py in the current directory and then import the module using the import command. You can then access the log_search functions from your main application module. We'll dive into modules and functions in more detail, but for now think of a module as a collection of definitions and statements that comprise a set of related capabilities. Each function within a module is accessed using the module name, followed by a period (.) like this:

import log_search

log_search.find("pythonstart.com")

What is a Python Package?

A package is a collection of Python modules. This lets you organize even larger sets of capabilities into a single package. For example, perhaps you want to create an log_analysis package that includes many different sets of capabilities, and perhaps it will grow over time. Within this package, you want to include a set of functions that let users search log files ( log_search ) and a set of functions to append, split, and save log files, which you'll call ( log_file_ops ). Each module is a file that sits within a log_analysis folder. You can even nest packages, so that you can include multiple packages in a single web_ops package. To do this, you simply create a top-level folder named web_ops , add the log_analysis folder and create sibling folders such as web_server . Below is a view of the folder structure:

web_ops/ Top-level package

__init__.py Initialize the web_ops package

web_server/ Subpackage for web server functionality

__init__.py Initialize the web_server package

deploy.py Functions to deploy your web server

start.py Functions to start your web server

log_analysis/ Subpackage for log analysis functionality

__init__.py Initialize the log_analysis package

log_search.py Functions to search your logs

log_file_ops .py Functions to modify your log files

Popular Python Packages

Below are a few popular Python packages as examples:

7 Brilliant Python Examples You Should Know

info@pythonstart.com (Python Start) — Sat, 25 Feb 2017 08:00:00 GMT

Python is a powerhouse. Here are 7 real-world examples to prove it.

Perhaps you have heard that Python is the programming language of choice for many developers, but perhaps you are wondering who really uses it and what they build with Python. Here are 7 examples of Python's impressive versatility:

Pinterest - Now a social media magnet, attracting Google Execs , Pinterest's humble start is rooted in Python. "When we first started building Pinterest, we used Python as our development language, which helped us build quickly and reliably. Over the years we built many tools around Python, including Pinball, MySQL_utils and pymemcache, as well as a set of libraries used daily for service communication and configuration management." - Shu Zhang , Pinterest engineer, Infrastructure ( Medium )
YouTube - YouTube has grown to become a household name and "its most famous star, PewDiePie , makes approximately $15 million a year through his following of more than 50 million, according to Forbes." ( The Witchita Eagle ) "Python is fast enough for our site and allows us to produce maintainable features in record times, with a minimum of developers." - Cuong Do, Software Architect, YouTube.com ( Python.org )
NASA - The TkInter module is a Graphical User Interface (GUI) toolkit for Python . TkInter was used extensively in NASA's Workflow Automation System (WAS). "Python allows us to tackle the complexity of programs like the WAS without getting bogged down in the language." - Robin Friedrich, Senior Project Engineer, United Space Alliance -- NASA's main shuttle support contractor. ( Python.org )
Industrial Light & Magic (Lucasfilm, Walt Disney) - ILM has forumlated billions of lines of code to support 300+ movies and Rogue One recently earned an Oscar nomination for its special effects. "Python plays a key role in our production pipeline. Without it a project the size of Star Wars: Episode II would have been very difficult to pull off. From crowd rendering to batch processing to compositing, Python binds all things together." - Tommy Burnette, Senior Technical Director, Industrial Light & Magic . ( Python.org )
Dropbox - Dropbox now has over half a billion users around the world. via Dropbox blog "[Python has] helped [Dropbox] stay lean and the product remain simple as it [scaled] to let over 100 million users save over 1 billion files per day." ( TechCrunch )
Google - "Python has been an important part of Google since the beginning, and remains so as the system grows and evolves. Today dozens of Google engineers use Python, and we're looking for more people with skills in this language." - Peter Norvig, director of search quality at Google, Inc. ( Python.org )
EA Games (Electronic Arts) - One of the reasons Battlefield 2 remained so popular , was its support for mods written in Python ( FZ ). Python has been used by game developers as early as Earth & Beyond . (Sean Riley's Game Programming With Python )

Do you know of other noteworthy Python examples? Please tweet them to us at @StartPython .

Getting Started with Python

info@pythonstart.com (Python Start) — Thu, 23 Feb 2017 08:00:00 GMT

Learning Python for APIs, analysis, automation, and deep learning

Python is growing in popularity and is currently listed among the top 5 programming languages in the TIOBE Index . Why is Python so popular? Below are 5 reasons:

Libraries - many popular and supported libraries (e.g., asyncio , matplotlib , numpy )
Easy to learn - Guido van Rossum created Python based on his experience with its precursor, ABC
Highly portable - Python is interpreted, so there's no need to compile binaries for every platform
Popularity for analytics - Python is seeing growing use for scholarly analytics
Readily available - you can download the latest version of Python for free

This blog is designed to help you get started learning Python for fun and for profit!