Contributing¶

Our project welcomes new contributors who want to help us fix bugs and improve our scrapers. The current status of our effort is documented in our sources list and issue tracker. We want your help. We need your help. Here’s how to get started.

Adding features and fixing bugs is managed using the GitHub’s pull request system.

The tutorial that follows assumes you have the Python programming language, the pipenv package manager and the git version control system already installed. If you don’t, you’ll want to address that first.

Create a fork ¶

Start by visiting our project’s repository at github.com/biglocalnews/warn-scraper and creating a fork. You can learn how from GitHub’s documentation.

Clone the fork ¶

Now you need to make a copy of your fork on your computer using GitHub’s cloning system. There are several methods to do this, which are coverd in the GitHub documentation.

A typical terminal command will look something like the following, with your username inserted in the URL.

git clone git@github.com:yourusername/warn-scraper.git

Install dependencies ¶

You should change directory into folder where you code was downloaded.

cd warn-scraper

The pipenv package manager can install all of the Python tools necessary to run and test our code.

pipenv install --dev

Now install pre-commit to run a battery of automatic quick fixes against your work.

pipenv run pre-commit install

Create an issue ¶

Before you being coding, you should visit our issue tracker and create a new ticket. You should try to clearly and succinctly describe the problem you are trying to solve. If you haven’t done it before, GitHub has a guide on how to create an issue.

Create a branch ¶

Next will we create a branch in your local repository where you an work without disturbing the mainline of code.

You can do this by running a line of code like the one below. You should substitute a branch that summarizes the work you’re trying to do.

git checkout -b your-branch-name

For instance, if you were trying to add a scraper for the state of Iowa, you might name your branch ia-scraper.

git checkout-b ia-scraper

We ask that you follow a pattern where the branch name includes the postal code of the state you’re working on, combined with the issue number generated by GitHub. In this example, work on the New Jersey scraper collected in GitHub issue #100.

git checkout -b nj-100

Perform your work ¶

Now you can begin your work. You can start editing the code on your computer, making changes and running scripts to iterate toward your goal.

Creating a new scraper ¶

When adding a new state, you should create a new Python file in the warn/scrapers named with the state’s postal code. Here is an example of a starting point you can paste in to get going.

from pathlib import Path

from .. import utils
from ..cache import Cache


def scrape(
    data_dir: Path = utils.WARN_DATA_DIR,
    cache_dir: Path = utils.WARN_CACHE_DIR,
) -> Path:
    """
    Scrape data from Iowa.

    Keyword arguments:
    data_dir -- the Path were the result will be saved (default WARN_DATA_DIR)
    cache_dir -- the Path where results can be cached (default WARN_CACHE_DIR)

    Returns: the Path where the file is written
    """
    # Grab the page
    page = utils.get_url("https://xx.gov/yy.html")
    html = page.text

    # Write the raw file to the cache
    cache = Cache(cache_dir)
    cache.write("xx/yy.html", html)

    # Parse the source file and convert to a list of rows, with a header in the first row.
    ## It's up to you to fill in the blank here based on the structure of the source file.
    ## You could do that here with BeautifulSoup or whatever other technique.
    pass

    # Set the path to the final CSV
    # We should always use the lower-case state postal code, like nj.csv
    output_csv = data_dir / "xx.csv"

    # Write out the rows to the export directory
    utils.write_rows_to_csv(output_csv, cleaned_data)

    # Return the path to the final CSV
    return output_csv


if __name__ == "__main__":
    scrape()

When creating a scraper, there are a few rules of thumb.

The raw data being scraped — whether it be HTML, CSV or PDF — should be saved to the cache unedited. We aim to store pristine versions of our source data.
The data extracted from source files should be exported as a single file. Any intermediate files generated during data processing should not be written to the data folder. Such files should be written to the cache directory.
The final export should be the state’s postal code, in lower case. For example, Iowa’s final file should be saved as ia.csv.
For simple cases, use a cache name identical to the final export name.
If many files need to be cached, create a subdirectory using the lower-case state postal code and apply a sensible naming scheme to the cached files (e.g. mo/page_1.html).

Here’s an example directory demonstrating the above conventions:

├── cache
│   ├── mo.csv
│   ├── nj
│   │   ├── Jan2010Warn.html
│   │   └── Jan2011Warn.html
│   ├── ny_raw_1.csv
│   └── ny_raw_2.csv
└── exports
    ├── mo.csv
    ├── nj.csv
    └── ny.csv

A fully featured example to learn from is Chris Zubak-Skees’ scraper for Georgia.

Running the CLI ¶

After a scraper has been created, the command-line tool provides a method to test code changes as you go. Run the following, and you’ll see the standard help message.

pipenv run python -m warn.cli --help

Usage: python -m warn.cli [OPTIONS] [SCRAPERS]...

  Command-line interface for downloading WARN Act notices.

  SCRAPERS -- a list of one or more postal codes to scrape. Pass `all` to
  scrape all supported states and territories.

Options:
  --data-dir PATH                 The Path were the results will be saved
  --cache-dir PATH                The Path where results can be cached
  --delete / --no-delete          Delete generated files from the cache
  -l, --log-level [DEBUG|INFO|WARNING|ERROR|CRITICAL]
                                  Set the logging level
  --help                          Show this message and exit.

Running a state is as simple as passing arguments to that same command. If you were trying to develop an Iowa scraper found in the warn/scrapers/ia.py file, you could run something like this.

pipenv run python -m warn.cli IA

For more verbose logging, you can ask the system to showing debugging information.

pipenv run python -m warn.cli IA -l DEBUG

You could continue to iterate with code edits and CLI runs until you’ve completed your goal.

Run tests ¶

Before you submit your work for inclusion in the project, you should run our tests to identify bugs. Testing is implemented via pytest. Run the tests with the following.

make test

If any errors, arise, carefully read the traceback message to determine what needs to be repaired.

Push to your fork ¶

Once you’re happy with your work and the tests are passing, you should commit your work and push it to your fork.

git commit -am "Describe your work here"
git push -u origin your-branch-name

If there have been significant changes to the main branch since you started work, you should consider integrating those edits to your branch since any differences will need to be reconciled before your code can be merged.

# Checkout and pull updates on main
git checkout main
git pull

# Checkout your branch again
git checkout your-branch-name

# Rebase your changes on top of main
git rebase main

If any code conflicts arise, you can open the listed files and seek to reconcile them yourself. If you need help, reach out to the maintainers.

Once that’s complete, commit any changes and push again to your fork’s branch.

git commit -am "Merged in main"
git push origin your-branch-name

Send a pull request ¶

The final step is to submit a pull request to the main respository, asking the maintainers to consider integrating your patch. GitHub has a short guide that can walk you through the process. You should tag your issue number in the request so that they linked in GitHub’s system.