\\nError!
\\nThe following unhandled error has occurred in the routine WLLQRYNA$LCEV2.ActionQuery:\\n
\\nORA-01403: no data found\\n
\\nPlease contact your support representative.\\n
\\n\\n\\n'"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"response.text"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you read closely, **that's an error**. It's because **we didn't send it any search data.**\n",
"\n",
"> Looking at `response.text` is THE BEST WAY to find out whether your search worked. You can ctrl+f or just visually search for words you know should be on the page."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Finding our form data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When we clicked \"Search,\" it also sent the server a bunch of data - all of the options we typed in, or the dropdowns we selected. Here are the steps to find out what data needs to be sent along with your request.\n",
"\n",
"We're going to use Chrome's **Network tools** to analyze all of the requests our browser sends to the server, then imitate them in Python.\n",
"\n",
"1. Open up **Developer Tools** in Chrome by selecting `View > Developer > Developer Tools`.\n",
"2. Select the **Network Tab**\n",
"3. Visit the page you're going to do your search from\n",
"4. Click the **Clear** button up top - 🚫 - then submit your form\n",
"5. The Network tab will fill with activity!\n",
"6. Find the thing in the Network tab that looks like the same name as your webpage. Click it.\n",
"7. On the right-hand side you get a new pane. If you scroll allllll the way down it lists **Form Data**.\n",
"\n",
"This **Form Data** is what we need to send along with our request. We just need to convert it to a dictionary and send it along."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Sending data with the form request\n",
"\n",
"Once we've converted our form data into a dictionary, we need to make sure of two things:\n",
"\n",
"1. We're using `requests.post` to make our request\n",
"2. We're sending the form data with the request\n",
"\n",
"Normal browser requests are sent as `GET` requests, but these very fancy ones are sent as `POST`. `POST` just means \"hey I'm sending extra data along with this.\""
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"NameTypeNumberStatusAddressCityZipCounty\n",
"SMITH A M OM2554CANCELLED2245 ASHBOURNE DRSAN MARINO91108LOS ANGELES\n",
"SMITH ALLEN EL2352CANCELLED713 N CALIFORNIA STBURBANK91505LOS ANGELES\n",
"SMITH ALVIN JE490DECEASED5004 RAMSDELL AVELA CRESCENTA91214LOS ANGELES\n",
"SMITH ARTHUR KERMITCS3124CANCELLED28803 CEDARBLUFF DRRANCHO PALOS VERDES90275LOS ANGELES\n"
]
}
],
"source": [
"data = {\n",
" 'P_QTE_CODE': 'ENG',\n",
" 'P_QTE_PGM_CODE': '7500',\n",
" 'P_LAST_NAME': 'smith',\n",
" 'P_FIRST_NAME': '',\n",
" 'P_INITIAL': '',\n",
" 'P_LICENSE_NUM': '',\n",
" 'P_CITY': '',\n",
" 'P_COUNTY': 'LOS ANGELES',\n",
" 'P_RECORD_SET_SIZE': '50',\n",
" 'Z_ACTION': 'Find'\n",
"}\n",
"\n",
"# Get the page\n",
"# use .post\n",
"# send the data\n",
"url = \"http://www2.dca.ca.gov/pls/wllpub/WLLQRYNA$LCEV2.ActionQuery\"\n",
"response = requests.post(url, data=data)\n",
"doc = BeautifulSoup(response.text, 'html.parser')\n",
"\n",
"# Grab all of the rows\n",
"row_tags = doc.find_all('tr')\n",
"\n",
"# Let's print the first 5\n",
"for row in row_tags[:5]:\n",
" print(row.text.strip())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If we didn't know if it worked or not, we could also check the response by looking at `response.text`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Sending headers with your request\n",
"\n",
"Sometimes that isn't enough! Some web servers check to make sure you're a real browser, or you came from their site, or other stuff like that.\n",
"\n",
"**We don't need to do this for the Engineers page, but I'm going to do it anyway.**\n",
"\n",
"When you send a request, you also send thing called \"Headers.\" You can see the headers inside of the same Network tab part where you found Form Data. It's listed as **Request Headers** - *ignore the response headers*.\n",
"\n",
"### Pretending to be the browser\n",
"\n",
"The most common thing you'll need to do is impersonate a browser by sending a `User-Agent` string. If we wanted to visit Columbia's website pretending to be Chrome, we might do this:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"