I know this is Code for Philly and (a) I am no longer in Philly and (b) this is not a Philly-specific project. Forgive me! Y’all are the people I know.
I am looking for someone who can help me (and others) export information from sites like AirBnB and VRBO into a shareable form so that it can be used to place Houston area residents who need temporary housing. It could be as simple as exporting to a Google doc or something more complicated that updates in real time.
The need is evolving, so there might be more specific requirements by tomorrow.
Let me know if you’re interested in thinking this through and/or if you have ideas of others outside of Philly who might.
I’m definitely interested in helping out with this. I have some experience in scraping sites for data like this and am confident that I can get the necessary data from both AirBnB and VRBO.
By sharable, do you mean offline sharable? If not offline, what makes this more useful than the end users searching directly on the AirBnB and VRBO sites?
Offline shareable, but ideally in a form that can be updated–both from the data source and the people accessing it.
The scenario is less individual people in need of homes connecting with individual homeowners and more a core group of support people working to place a large group of people, many of whom won’t have internet access. Does that make sense?
The relevant data that comes to mind (there might be more):
Availability
home or private room
Location
Price
Made available for free to Harvey evacuees (folks are opting in)
Number of beds
Number of guests allowed
Contact info? (Airbnb might not allow)
On the input side, from the other end, some way to update status (inquiry made, pending response, evacuees placed, etc.)
I’ll get started on putting together the scraper now. Starting with AirBnB. I’ll try to make it as flexible as I can quickly so we can add additional needed data parameters later if need be.
Thanks! I’m actually not sure how to tag Jason here, but hopefully he will see this, too! I’m trying to get sense of how long or short/complicated or uncomplicated this is. The folks in Austin have 200 people arriving now through tonight and are asking if I can just use Google Forms/Sheets to do something quickly. Depending on what Jason’s been able to do, maybe that’s not necessary?
Let’s definitely get a sense of what Jason’s accomplished so far. I’m a little unclear on criteria here - so far I understand that we’d like to:
Scrape or collect temporary housing options from sites like AiBnB and VRBO
Serve that content in some downloadable offline form that ideally could be updated in real time
Questions:
What’s the criteria for selecting data from those sites?
What’s the geographical range for housing? It kinda seems like AirBnB places in Houston will be largely out of commission for a while.
Are there any TOS violation issues with scraping this stuff from AirBnB etc? I’d hate to collect that information with the best intentions, only to be banned or hindered in some way because we broke rules.
I know that lots of folks have scraped the data before in order to do analysis on policy issues related to affordable housing availability. but I have not read the TOS. Maybe you can take a look? I am doing this for folks in Austin who are receiving people now.
What do you mean when you say criteria for selecting data? I think I know what you mean, but I don’t want to answer the wrong question.
Sure, happy to take a look. And I just mean, what’s the best information to provide people with? Austin-area places where people can seek shelter for free, or Houston-area places that are under a certain price, etc. Just trying to clarify what kind of info we’re trying to connect people with to make we serve them as well as we can.
And hey, no worries if you answer the wrong question! It will probably still be a useful answer to work with.
I’ve got the AirBnB scraper working now. Geographic range is going to be useful as I need to define which area is being scraped. AirBnB limits to 300 listings per area selected.
Can you start up a Google Sheet and share with us that I can dump the data into. We can determine best route of handling the data once we can all view it. I’m currently dumping the data in JSON format and converting it to CSV manually. Can look into doing this in real-time once we get this working they way we want as far as getting the proper data out there.
Hey folks, I put a bunch of feelers out, this morning, and a guy just came through and said he was able to convert the data to csv. (Sorry sorry sorry! I was trying not to have folks running on parallel tracks, but it just happened.). If I am able to share that with you, would you be interested in working on the Lat/Long mapping issue? I need to confirm with folks on the ground, but I think it would be real helpful.
No problem! The easiest way to handle the mapping issue would be to use Google Maps’ built-in import tool. I’m not sure how well that will work with a large dataset, though, but I’m happy to give it a try and see.
Hey everyone, just seeing this now–I have an idea. Laura, can we connect with the guy who did the csv conversion–and learn how he did it? Then I can add a readme to the GitHub phase so that this can easily be done again and again–since the data will change as new places are added, etc. We can make this work that’s been done useful over time–and in other places.
I was thinking about “cleaning” the data as it’s scraped and dump right into CSV as well. Could probably remove some of the unnecessary pieces of data as well, since the amount of data being dumped is quite large.