ISO of tech savvy to support displaced Houston residents

Help Wanted
#21

Hi folks!

Thanks for continue to think this through. I just emailed Chad the link to this convo and to ask whether he’d rather connect here or through email.

I have a .csv file that’s limited to the necessary pieces of data. Can I upload here? I’m not seeing the way to do that. I can also create a clean Google sheet with the info: https://docs.google.com/spreadsheets/d/17nUTv7cOPOkEo9WmkIApQU1nDyfsEXW_3-jg_l5Re4c/edit?usp=sharing

Preston said he’d be interested in working on mapping, but couldn’t post beyond 3X because he’s new to Code for Philly (why is that a thing?).

#22

Hey there all. The coding yesterday to get the CSV was pretty quick and dirty, so would need some massaging to be put into a workable format. I did it in ColdFusion just because I had some similar code on a client’s server already written to get it done quicker, but I could migrate the code to PHP.

I used this http://airbnbapi.com primarily the “View Listing Info”. I had hoped that maybe the “Listing Search” would work via the “disaster_id” that AirBnB posted, or the $0 or something, but after a few unsuccessful guesses and tries I just moved forward.

I started with a scrape of the listing ID from the HTML code on all 35 pages, literally just loading that HTML, looking for the “data-url” of “/rooms/XXXX” and collected a list of all those IDs.

I should say here that while we can work fast doing things this way, it’s definitely worth trying to engage AirBnB directly too to see if there’s a better way to get this data, and it’s also possible there’s a way to use the API to grab that list directly.

Anyway, then looping through that ID list, called the API for the listing info, it returns the full JSON for that listing, example here:

https://api.airbnb.com/v2/listings/326936?client_id=3092nxybyb0otqw18e8nh5nty&_format=v1_legacy_for_p3

Then just parsed out the fields necessary, putting together a few things dynamically…the URLs, checking for pets, etc.

If this is going to be something to continue using, and also if AirBnB can’t provide the information directly then it’d definitely make sense to systemize it, have the parsed data saved into a database instead, process to regularly re-scape, check for new IDs, etc.

I’m happy to provide whatever I can, just let me know how you’re working, what you’d need me to do, etc. There are some things with the API to check, if there are any limitations with the client ID being used there, and then for more complex things like pulling calendar data, an access token is also necessary, and I haven’t tried doing that to see what’s returned. Again, did it very quick and dirty so would need to check more of that.

1 Like
#23

Nice Chad, I didn’t know about that API being publicly accessible. I’ve updated my code quite a bit to scrape the listing ID’s, then use the API to gather all the listing info. It’s running a lot faster now. I’ve also updated to dump directly into a CSV file. You can check the github page for the latest update to this. Below is updated CSV file as of about 5pm EST. Didn’t want to overwrite the google doc just yet.

It would be nice to get this running and updating the spreadsheet automatically. What’s the best way of doing this? I don’t have a lot of experience with Google Docs. At the least I can set this up online and have it email the CSV to someone responsible for uploading? Or add a web interface to download the latest copy on demand?

Also, on a side note, AirBnB list contains availability for 1,048 beds and 150 (26%) of listings allow pets.

1 Like
#24

Awesome work, Jason! We def don’t want to override the Google Sheet b/c we formatted it to work for the volunteers and they are adding info. (Google Sheets is subpar for mobile and tablets, so I had to do manual linebreaks and intersperse the headings, blah blah.)

I just reached out to one of the people on the ground to ask whar she’d like me to do. Most likely, I need to upload from csv into a new Google Sheet and then combine the two. I’m in transit now, but will follow up when I’ve heard from the Austin crew

#25

GREAT! The listings (rooms) have a unique ID, so it would technically be possible to only add records which aren’t already in there. I’m not sure if Google sheets allow you to do all the things that Excel can, but you can do compares of two sheets based on an ID column in Excel, only get the new ones, etc, but that’s all someone knowing how to do that from two different files to compare them. My expertise is on the web app side, online database, etc.

I think that yes, setting it up online so that someone can hit a page to run and get a full new CSV is a good step and then folks can manipulate things in Excel.

Ultimately if there’s a longer term need, and also if we’re looking to add additional sources, setting it up to save in a database would make sense. Then it first checks to see if the listing exists, if so, ignore it, otherwise insert it. Options to return only new data since last time, or full data set, etc. We quickly get into a bigger, but completely doable project.

#26

Excellent, I downloaded the new csv and will merge (or whatever) with the excel in the morning. Hoping to get more clarity about next steps and what’s needed, tomorrow. For the moment, I know that folks are using what we’ve provided, but I’m not hearing much else back. I’ll keep you awesome people posted.

#27

OK. Looking at the new csv file now. Jason’s new csv file has different data entries than Chad’s, some of the new data being very useful (notably, last calendar and property type). However, I am not seeing the unique ID, which is the thing that should facilitate merging of the two excel spreadsheets. Jason, are you able to get access to a version with the unique id?

#28

Check out this project: https://github.com/sketch-city/harvey-api

Folks are working on redeploying it in Florida.

There is a national slackline room connected to Code for Philly’s slack at #disaster-redeployment

There’s also a whole slack team for irma responses by civic hackers that you can join here: https://join.slack.com/t/irma-response/shared_invite/MjM3NzI4NDM2OTUxLTE1MDQ2NDc0OTQtZmU2YWUxOGJmYg