| CARVIEW |
Select Language
HTTP/2 200
date: Fri, 25 Jul 2025 08:04:50 GMT
content-type: text/html; charset=utf-8
vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, X-Requested-With,Accept-Encoding, Accept, X-Requested-With
x-robots-tag: none
etag: W/"a6bcbb14d9df1d0a3290a30f4c32f740"
cache-control: max-age=0, private, must-revalidate
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 0
referrer-policy: no-referrer-when-downgrade
content-security-policy: default-src 'none'; base-uri 'self'; child-src github.githubassets.com github.com/assets-cdn/worker/ github.com/assets/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com www.githubstatus.com collector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com *.rel.tunnels.api.visualstudio.com wss://*.rel.tunnels.api.visualstudio.com objects-origin.githubusercontent.com copilot-proxy.githubusercontent.com proxy.individual.githubcopilot.com proxy.business.githubcopilot.com proxy.enterprise.githubcopilot.com *.actions.githubusercontent.com wss://*.actions.githubusercontent.com productionresultssa0.blob.core.windows.net/ productionresultssa1.blob.core.windows.net/ productionresultssa2.blob.core.windows.net/ productionresultssa3.blob.core.windows.net/ productionresultssa4.blob.core.windows.net/ productionresultssa5.blob.core.windows.net/ productionresultssa6.blob.core.windows.net/ productionresultssa7.blob.core.windows.net/ productionresultssa8.blob.core.windows.net/ productionresultssa9.blob.core.windows.net/ productionresultssa10.blob.core.windows.net/ productionresultssa11.blob.core.windows.net/ productionresultssa12.blob.core.windows.net/ productionresultssa13.blob.core.windows.net/ productionresultssa14.blob.core.windows.net/ productionresultssa15.blob.core.windows.net/ productionresultssa16.blob.core.windows.net/ productionresultssa17.blob.core.windows.net/ productionresultssa18.blob.core.windows.net/ productionresultssa19.blob.core.windows.net/ github-production-repository-image-32fea6.s3.amazonaws.com github-production-release-asset-2e65be.s3.amazonaws.com insights.github.com wss://alive.github.com api.githubcopilot.com api.individual.githubcopilot.com api.business.githubcopilot.com api.enterprise.githubcopilot.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com copilot-workspace.githubnext.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src viewscreen.githubusercontent.com notebooks.githubusercontent.com; img-src 'self' data: blob: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com avatars.githubusercontent.com private-avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com release-assets.githubusercontent.com secured-user-images.githubusercontent.com/ user-images.githubusercontent.com/ private-user-images.githubusercontent.com opengraph.githubassets.com copilotprodattachments.blob.core.windows.net/github-production-copilot-attachments/ github-production-user-asset-6210df.s3.amazonaws.com customer-stories-feed.github.com spotlights-feed.github.com objects-origin.githubusercontent.com *.githubusercontent.com; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/ secured-user-images.githubusercontent.com/ private-user-images.githubusercontent.com github-production-user-asset-6210df.s3.amazonaws.com gist.github.com; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; upgrade-insecure-requests; worker-src github.githubassets.com github.com/assets-cdn/worker/ github.com/assets/ gist.github.com/assets-cdn/worker/
server: github.com
content-encoding: gzip
accept-ranges: bytes
set-cookie: _gh_sess=7jCsgk3g%2FjgmGLlizl9ZCFZOYO8QFd7KXRrlXx5mZ5qPX1fHM3YZNlAkVEg6LVrq61Hw2Zeh0LodCHLxRkXfAB01GNAULeS%2BBqMr1gikMAdcvXw1A4RGT2od388B%2FnsI%2FJOgpsXmmBKo8O5m8j5NUsVrRiskJRPBtD2lhoXZYW2%2BiRVM%2BJN1VuVlSrMiIg%2FK7vkqFBB8xX7QOXJYX9MiVMvVzvB1DoaOnjYjmPkVWu02oIHHvzsvkgSiXeqY2Ax6cRAv%2F5y0zoVc7%2FDC8lj2XA%3D%3D--jFT%2Fgk%2F%2Bf8s0iI8i--zlT7VUD%2BoQ5SWWbv4FPXRQ%3D%3D; Path=/; HttpOnly; Secure; SameSite=Lax
set-cookie: _octo=GH1.1.386770708.1753430690; Path=/; Domain=github.com; Expires=Sat, 25 Jul 2026 08:04:50 GMT; Secure; SameSite=Lax
set-cookie: logged_in=no; Path=/; Domain=github.com; Expires=Sat, 25 Jul 2026 08:04:50 GMT; HttpOnly; Secure; SameSite=Lax
x-github-request-id: BB6C:9D20F:5048F4:611C18:68833AA2
Script: pcurl.sh · timrdf/csv2rdf4lod-automation Wiki · GitHub
Skip to content
Navigation Menu
{{ message }}
-
Notifications
You must be signed in to change notification settings - Fork 36
Script: pcurl.sh
Tim L edited this page Feb 4, 2014
·
45 revisions
csv2rdf4lod-automation is licensed under the [Apache License, Version 2.0](https://github.com/timrdf/csv2rdf4lod-automation/wiki/License)
- pcurl.sh can and should be used in Conversion process phase: retrieve
pcurl.sh is one of the most essential scripts for providing transparency in csv2rdf4lod.
While the rest of csv2rdf4lod-automation is dedicated to converting and publishing well-structured, highly connected RDF from tabular data, pcurl.sh captures the implicit connection from all of our local processing results to the original data provided by a more authoritative source organization. Associating our local results to the original data source enables accountability, repeatability, and attribution -- essential aspects for establishing trust in our third party enhancements. pcurl.sh helps us fulfill our Design Objective: Capturing and Exposing Provenance.
Conversion process phase: retrieve also shows an example for how pcurl.sh is used.
Script location: $CSV2RDF4LOD_HOME/bin/pcurl.sh
bash-3.2$ pcurl.sh
usage: pcurl.sh [-I] url [-n name] [-e extension] [url [-n name] [-e extension]]*
-I : do not download file; just obtain HTTP header information (c.f. curl -I)
url : the URL to retrieve
-n : use 'name' as the local file name.
-e : use 'extension' as the extension to the local file name.
pcurl.sh https://www.whitehouse.gov/files/disclosures/visitors/WhiteHouse-WAVES-Released-0111.zip
creates WhiteHouse-WAVES-Released-0111.zip and WhiteHouse-WAVES-Released-0111.zip.pml.ttl in the current working directory.
The remaining blocks of PML encoded in Turtle are one continuous provenance capture of the URL retrieval shown above, with comments describing the subsequent representation.
@prefix rdfs: <https://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <https://www.w3.org/2001/XMLSchema#> .
@prefix dcterms: <https://purl.org/dc/terms/> .
@prefix pmlp: <https://inference-web.org/2.0/pml-provenance.owl#> .
@prefix pmlj: <https://inference-web.org/2.0/pml-justification.owl#> .
@prefix irw: <https://www.ontologydesignpatterns.org/ont/web/irw.owl#> .
@prefix nfo: <https://www.semanticdesktop.org/ontologies/nfo/#> .
@prefix conv: <https://purl.org/twc/vocab/conversion/> .
@prefix httphead: <https://inference-web.org/registry/MPR/HTTP_1_1_HEAD.owl#> .
@prefix httpget: <https://inference-web.org/registry/MPR/HTTP_1_1_GET.owl#> .
The URL from which we retrieved our file, with a modification date reported by the HTTP server:
<https://www.whitehouse.gov/files/disclosures/visitors/WhiteHouse-WAVES-Released-0111.zip>
a pmlp:Source;
.
<https://www.whitehouse.gov/files/disclosures/visitors/WhiteHouse-WAVES-Released-0111.zip>
a pmlp:Source;
pmlp:hasModificationDateTime "2011-01-28T23:19:12"^^xsd:dateTime;
.
The file on our local disk, which we md5 hashed:
<WhiteHouse-WAVES-Released-0111.zip>
a pmlp:Information;
pmlp:hasReferenceSourceUsage <sourceUsage_4921a599-96e2-46a3-bb6e-e6769d4b3fdf_content>;
nfo:hasHash <md5_b76602e45b2e9a76869200b877d01f1c>;
.
<md5_b76602e45b2e9a76869200b877d01f1c>
a nfo:FileHash;
nfo:hashAlgorithm "md5";
nfo:hasHash "b76602e45b2e9a76869200b877d01f1c";
.
Justifying the existence of our file on disk as the result of curl HTTP requesting the file:
<nodeSet_4921a599-96e2-46a3-bb6e-e6769d4b3fdf_content>
a pmlj:NodeSet;
pmlj:hasConclusion <WhiteHouse-WAVES-Released-0111.zip>;
pmlj:isConsequentOf [
a pmlj:InferenceStep;
pmlj:hasIndex 0;
pmlj:hasAntecedentList ();
pmlj:hasSourceUsage <sourceUsage_4921a599-96e2-46a3-bb6e-e6769d4b3fdf_content>;
pmlj:hasInferenceEngine conv:curl_md5_5670dffdc5533a4c57243fc97b19a654;
pmlj:hasInferenceRule httpget:HTTP_1_1_GET;
];
.
The time that we retrieved the file:
<sourceUsage_4921a599-96e2-46a3-bb6e-e6769d4b3fdf_content>
a pmlp:SourceUsage;
pmlp:hasSource <https://www.whitehouse.gov/files/disclosures/visitors/WhiteHouse-WAVES-Released-0111.zip>;
pmlp:hasUsageDateTime "2011-02-22T22:30:42-05:00"^^xsd:dateTime;
.
The header information of the HTTP retrieval:
<info_4921a599-96e2-46a3-bb6e-e6769d4b3fdf_url_header>
a pmlp:Information, conv:HTTPHeader;
pmlp:hasRawString """HTTP/1.1 200 OK
ETag: "b76602e45b2e9a76869200b877d01f1c:1296256752"
Last-Modified: Fri, 28 Jan 2011 23:19:12 GMT
Accept-Ranges: bytes
Content-Length: 1247427
Content-Type: application/zip
Date: Wed, 23 Feb 2011 03:30:41 GMT
Connection: keep-alive
Server: White House
P3P: CP="NON DSP COR ADM DEV IVA OTPi OUR LEG"
""";
pmlp:hasReferenceSourceUsage <sourceUsage_4921a599-96e2-46a3-bb6e-e6769d4b3fdf_url_header>;
.
Justifying the HTTP header as the result of curl requesting an HTTP HEAD:
<nodeSet_4921a599-96e2-46a3-bb6e-e6769d4b3fdf_url_header>
a pmlj:NodeSet;
pmlj:hasConclusion <info_4921a599-96e2-46a3-bb6e-e6769d4b3fdf_url_header>;
pmlj:isConsequentOf [
a pmlj:InferenceStep;
pmlj:hasIndex 0;
pmlj:hasAntecedentList ();
pmlj:hasSourceUsage <sourceUsage_4921a599-96e2-46a3-bb6e-e6769d4b3fdf_url_header>;
pmlj:hasInferenceEngine conv:curl_md5_5670dffdc5533a4c57243fc97b19a654;
pmlj:hasInferenceRule httphead:HTTP_1_1_HEAD;
];
.
The time that we retrieved the HTTP HEAD:
<sourceUsage_4921a599-96e2-46a3-bb6e-e6769d4b3fdf_url_header>
a pmlp:SourceUsage;
pmlp:hasSource <https://www.whitehouse.gov/files/disclosures/visitors/WhiteHouse-WAVES-Released-0111.zip>;
pmlp:hasUsageDateTime "2011-02-22T22:30:42-05:00"^^xsd:dateTime;
.
Identifying the curl implementation that performed the retrievals for us:
conv:curl_md5_5670dffdc5533a4c57243fc97b19a654
a pmlp:InferenceEngine, conv:Curl;
dcterms:identifier "md5_5670dffdc5533a4c57243fc97b19a654";
dcterms:description """curl 7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: GSS-Negotiate IPv6 Largefile NTLM SSL libz """;
.
conv:Curl rdfs:subClassOf pmlp:InferenceEngine .
Invoking
pcurl.sh https://www.data.gov/download/1554/csv -n 1554 -e csv
results in this PML, which is illustrated in this pdf diagram (NOTE: pre-attribution illustration).
Note the provenir:, hartigprov: and oprov: attributes associating the InferenceStep to a user account, which in turn is related to a Person.
(Note: using VSR, which isn't published yet)
pcurl.sh `vsr-query-redirect-beginning-sources.sh ../../2010-10-31/source/sparql-service-description.ttl.pml.ttl` -e ttl
The curl command:
curl https://www.epa-echo.gov/cgi-bin/effluentdata.cgi \
-F "permit=NY0261343" -F "hits=1" > NY0261343.csv
can be performed using pcurl.sh:
pcurl.sh https://www.epa-echo.gov/cgi-bin/effluentdata.cgi \
-F "permit=NY0261343" -F "hits=1" -n NY0261343 -e csv
and captures the POST fields and values at pmlj:hasVariableMapping:
<inferenceStep_d159b3fc-7c38-44e1-b68d-b1d591385d68_content>
a pmlj:InferenceStep;
pmlj:hasIndex 0;
pmlj:hasAntecedentList ();
pmlj:hasSourceUsage <sourceUsage_d159b3fc-7c38-44e1-b68d-b1d591385d68_content>;
pmlj:hasInferenceEngine conv:curl_md5_5670dffdc5533a4c57243fc97b19a654;
pmlj:hasInferenceRule httppost:HTTP_1_1_POST;
oboro:has_agent <https://tw.rpi.edu/web/inside/machine/lebot_macbook#lebot>;
hartigprov:involvedActor <https://tw.rpi.edu/web/inside/machine/lebot_macbook#lebot>;
pmlj:hasVariableMapping [ pmlj:mapFrom "permit"; pmlj:mapTo "NY0261343"; ];
pmlj:hasVariableMapping [ pmlj:mapFrom "hits"; pmlj:mapTo "1"; ];
.
- bin/util/punzip.sh will extract files from a compressed file and include provenance for each file extracted.
- Script: pvload.sh uses pcurl.sh when it retrieves a URL to load into a SPARQL endpoint.
- Conversion process phase: csv ify describes how to use justify.sh to add provenance assertions for manual edits of a file.
Clone this wiki locally
You can’t perform that action at this time.