[go: up one dir, main page]

Skip to content

A Javascript module to scrape, analyze, and cache Congressional bills

License

Notifications You must be signed in to change notification settings

jeffreyshen19/BillScraper.js

Repository files navigation

BillScraper.js 📝📝📝

Build Status codecov npm version Inline docs License: CC0-1.0

A NPM module to scrape, analyze, and cache Congressional bills. Works for bills from the 113th, 114th, and 115th congress.

Table of Contents

  • docs: Holds markdown files that explain the project in more detail.
  • examples: Holds example code.
  • example_outputs: Holds example outputs.
  • lib: Holds all Javascript files
  • misc: Hold all graphics for the README.md
  • test: Holds all unit tests.

Getting Started

  • To get started, visit GET_STARTED.md.
  • For examples of how to use the library, visit examples.
  • A variety of guides on how to use the library are available in docs.
  • Documentation for all the methods is available here.

Why?

The government hosts bulk data for all congressional bills at gpo.gov/fdsys/bulkdata. However, this repository is horrible to use for developers, as bills are uploaded to unpredictable URLS as XML files. Each bill file is too structured, as XML style elements like paragraphs, bold text, and quotes are preserved, making it hard to parse the actual text of a bill.

What does this project do

  • Returns full Congressional bill text and metadata from Congress.gov as JSON. This library can scrape both the actual bill text (bills) and meta information such as cosponsors, amendments, and committees (bill statuses). For example, the following code will convert the XML (on the top) into the JSON (on the bottom)
var billScraper = require("bill-scraper");

billScraper.getBill("BILLS-113hr1033rs", function(res){
  billScraper.printBill(res, function(parsedResult){
    console.log(JSON.stringify(parsedResult));
  });
}, {session: 2});
XML JSON
XML JSON

For more detailed examples, visit examples/get_bills.js.

  • Parses bills for key points and ideas: This project generates "tags" for bills, allowing them to more easily searched. For instance, searching for all bills tagged with "battlefield".
var billScraper = require("bill-scraper");

billScraper.getBill("BILLS-113hr1033rs", function(res){
 billScraper.printBill(res, function(parsedResult){
   billScraper.tagBill(parsedResult, function(tags){
      console.log(tags);
   });
 });
}, {session: 2});

//Will return ['battlefield', 'sites','war','section','land','subsection','period','funds','appropriation','protection','revolutionary','public','sub','term','nations','civil','preservation','service','sellers','prohibition','purposes','person','entity','matter','influence','law','ratification','policy'];

For more detailed examples, visit examples/tag_bills.js.

  • Generates a searchable Mongo database of bills: For example, the following code will locally cache a database (see photo below) of parsed, searchable bills.
var billScraper = require("bill-scraper");
var mongoose = require("mongoose");

mongoose.connect("mongodb://localhost/bills");
billScraper.generateDB("hr", 115, 1);

Database

For documentation on the Bill struct stored in the database, visit BILL_TEMPLATE.md in docs, for more detailed examples, visit examples/database.js, and for information on searching bills, see the examples in examples/search_bills.js.

Community

This repository encourages all users to contribute. Please read the Code of Conduct and Contributing.md. For things to contribute, check out the issues or TODO.md.

Datasets used

Relevant libraries used

View the package.json for the full list of dependencies.

Contributors

View the full list of contributors here.

License

This repository is licensed as CC0. See the LICENSE for more information.


CC0
To the extent possible under law, Jeffrey Shen has waived all copyright and related or neighboring rights to BillScraper.js. This work is published from: United States.