| CARVIEW |
Explicit file encodings using click.File
I wanted to add a --encoding option to sqlite-utils insert which could be used to change the file encoding used to read the incoming CSV or TSV file - see sqlite-utils #182.
Just one problem: the Click file argument was defined using click.File(), which has useful features like automatically handling - for standard input. The code looked like this:
@click.argument("json_file", type=click.File(), required=True)
# ...
def command(json_file):
# ...click.File() takes an optional encoding= parameter, but that requires you to know the file encoding in advance. In my case I wanted to use the encoding optionally provided by an --encoding= option.
The workaround I came up with was to switch to using click.File("rb"), which opened the incoming file (or stdin stream) in binary mode. I could then wrap it in a codecs.getreader() object that could convert those bytes into a Python string using the user-supplied encoding.
@click.argument("json_file", type=click.File("rb"), required=True)
@click.option("--encoding", help="Character encoding for input, defaults to utf-8")
def command(json_file, encoding):
encoding = encoding or "utf-8"
json_file = codecs.getreader(encoding)(json_file)
reader = csv.reader(json_file)
# ...Related
- python Using io.BufferedReader to peek against a non-peekable stream - 2021-02-15
- sqlite Fixing broken text encodings with sqlite-transform and ftfy - 2021-01-18
- pytest Testing a Click app with streaming input - 2022-01-09
- linux Using iconv to convert the text encoding of a file - 2022-06-14
- python Running PyPy on macOS using Homebrew - 2022-09-14
- sqlite Importing CSV data into SQLite with .import - 2021-07-13
- sqlite Saving an in-memory SQLite database to a file in Python - 2023-04-08
- python Understanding option names in Click - 2020-09-22
- bash Finding CSV files that start with a BOM using ripgrep - 2021-05-28
- sqlite SQLite BLOB literals - 2020-07-29
Created 2020-10-16T13:39:08-07:00, updated 2020-10-16T16:15:49-07:00 · History · Edit