Archiving Minneapa

Or, for those not from this part of science-fiction fandom, just think of it as some rather challenging scanning and OCR issues. (Read about APAs.)

I sorted through three boxes from upstairs and got this:

The first three banker’s boxes, plus extras
Back cover on top of the stack of extras

And there are four more boxes up there waiting.

Now, there is probably a lot of duplication (mine plus Pamela’s copies).

Minn-StF owns an Epson duplex auto-feed scanner, which is kind of tailor-made for this job (“duplex” means it scans both sides of the sheet in one pass through). And it’s amazing how good we’ve gotten at handling individual sheets of paper using just a few plastic rollers. Still, when the paper is 40 or so years old and the stack includes many different kinds of paper intermixed, it can be a challenge. (Most Minneapas had at least offset paper, mimeo paper, ditto paper, and often twilltone. And the covers are sometimes card stock.) Luckily, restarting after a jam is easy, so long as you didn’t let it reset the page numbering to 1 automatically.

I made some test scans at 300 dpi and 400 dpi, and tried saving them as JPEG and TIFF files. The scanner was nearly twice as fast at 300 dpi than at higher resolutions, so I left resolution there. I was pleased, though a bit surprised, to find essentially no visible JPEG artifacts (at 80% quality) on all this text. You’re seeing a lot of the paper texture at full res, and it’s enough to satisfy the OCR software…and the JPEG file is 2 MB or less, the TIFF is about 34 MB. So I actually stored the images as JEPGS.  (Nearly 5000 pages from Saturday’s session, looks like; which was one of those three banker’s boxes.)

Many pages show some browning around the edges. It’s interesting how much variation there is among the different kinds of paper people used.

The print density and clarity varied quite a lot to begin with, as I remember. It certainly varies a lot today.  Here are some examples at 100% size.

OCR of this sort of material ranges from chancy to hopeless. The volume involved is such that no real quality control or proofreading pass on the OCR is possible, either. However, by using a clever PDF feature we can produce “PDF/A” files which, when opened, show you the image of the page, but when searched by the computer let it search the OCR output (including the images of every page does make the files big, though). Even when OCR is bad, it catches words correctly a lot of the time, so searching for a name or a topic keyword will find you many of the references. And important words in a discussion tend to be repeated, so you’ll be brought to most of the pages the discussion occurs on. (My OCR work on this is being done with an old version of ABBYY Finereader.)

There are legal and privacy issues that make it unlikely that the collection of scans will be posted publicly. They may well be available to people who were in Minneapa. Scanning them gives us backup copies and protection against further deterioration, and some convenience for some people with access to the collection.

Anyway…17 down, 383 to go!

Mobile Device Connectivity Speed

Mostly grousing about my specific situations.

Turns out my phone (LG G4) connects to my home WIFI at around 50 Mb/sec; pretty pitiful (and I don’t know how to get the phone to tell me what speed it connected at; that’s measured at Ookla; but my household WiFi is enough faster than that that it’s going to be reasonably accurate). In theory I should be able to get 20 times that, but I think the phone doesn’t support high enough WiFi specs.

Then, when I turn on VPN (OpenVPN, TUN, TCP) terminating at my router, that drops again, to maybe 7 Mb/sec.  However, I can still get that 7Mb/sec from the local library, which blocks most types of VPN connection (no idea why; but this configuration plus using port 443 makes it look essentially identical to an HTTPS browser connection, so few people block it). (The default VPN on the phone is faster, 30 Mb/sec or some such, but using 40-bit encryption or something really embarrassing like that. That one is also blocked at the library.)

Even the laptop connect seems slow; 300 Mb/sec, on a brand-new laptop that says it supports 801.11ac and an ASUS RT-AC66U which says it supports AC.  (This is sitting in the same room with, or on the opposite side of the wall from, where the WiFi hangs on the wall.)  For that matter the laptops reports an AC connection. I’ve ended up buying an Amazon dongle that plugs into the USB-C port and gives me three conventional USB 3.1 ports and a gigabit Ethernet port.

 

Cygwin protection issues accessing SAMBA shares

I’ve posted about my instance of this to the Cygwin mailing before, years ago and again just now, and read a lot of what’s online about it. I’ve also asked people over in the FreeNAS community (and gotten basicaly the same answer). Basically everything I can find says that the information is here.

What’s there doesn’t work for me. Across many Cygwin installs on multiple windows versions (at least 7 and 10) on at least 4 different hardware platforms. Including one work computer accessing a Synology server, so *completely* different software on the server side (dev network at work, no Active Directory there, so it’s just like home, local logons only).

The Samba server at home is a FreeNAS box. It’s *not* joined to any domain, nor are the windows boxes (it’s at home, I have no AD server, it’s all local logons). Currently running FreeNAS 11.1 U5, if it matters (latest), which seems to be running samba version 4.7.0-GIT-de2f31198c7-FreeNAS.

Details will be from my Windows 10 desktop, where I did a clean install of Cygwin-64 last night; it identifies itself (uname -a) as CYGWIN_NT-10.0 DDB4 2.10.0(0.325/5/3) 2018-02-02 15:16 x86_64 Cygwin.

This box has accessed this server (multiple FreeNAS and hence Samba servers over the years) as both Windows 7 and windows 10. It works fine in windows, I can adjust file security through the Windows explorer dialogs, etc. (My desktop computers, or my last 3 or 4 desktop coumpters really, running windows and Cygwin in some version, has accessed a file server via CIFS for most of my file access since about 2006; that fileserver has been Solaris with ZFS and then FreeNAS with ZFS. I’ve also had at least three laptops configured to use this server and having Cygwin, and they all behaved the same as the desktop at that moment. I started having protection problems when Cygwin made the changes described in the link above.)

Cygwin works fine on locally-hosted files (not that I have many; a small SSD for software installation, plus external drives I may attach from time to time, everything important lives on the fileserver). The protections Cygwin shows for files on the NAS look like what I will get if the underlying problem *is* something related to the ntsec article above. That kinda gives me hope that I’m just doing something wrong that I’m unable to spot.

On the FreeNAS box, user ddb (uid 1001) owns the files in question:

[root@fsfs /mnt/zp1/ddb/Documents/Recipes]# id ddb uid=1001(ddb) gid=1001(ddb) groups=1001(ddb),0(wheel),20(staff),1004(public),1007(music),1712(bdr)
[root@fsfs /mnt/zp1/ddb/Documents/Recipes]# ls -l S*
-rwxrwxr-x+ 1 ddb ddb 9605 May 30 2004 Sacher.asc
-rwxrwxr-x+ 1 ddb ddb 9600 May 30 2004 Sacher.doc
-rwxrwxr-x+ 1 ddb ddb 4867 May 30 2004 Salsa.asc
-rwxrwxr-x+ 1 ddb ddb 4864 May 30 2004 Salsa.doc
-rwxrwxr-x+ 1 ddb ddb 2181 May 30 2004 Shrmpstr.asc
-rwxrwxr-x+ 1 ddb ddb 2176 May 30 2004 Shrmpstr.doc
-rwxrwxr-x+ 1 ddb ddb 20841 Dec 4 2012 Spaghetti.odt

Locally, it’s mapped as Windows drive P: (at the ‘Documents’ level in the above path), but also directly accessible as //fsfs/ddb/Documents.

$ id
uid=197612(ddb) gid=545(Users) groups=545(Users),197121(None),197613(fsfsddb),4(INTERACTIVE),66049(CONSOLE LOGON),11(Authenticated Users),15(This Organization),113(Local account),66048(LOCAL),262154(NTLM Authentication),401408(Medium Mandatory Level)

$ ls -l /cygdrive/p/Recipes/S*
-rwxrwxr-x 1 Unknown+User Unix_Group+1001 9.7k May 30 2004 /cygdrive/p/Recipes/Sacher.asc
-rwxrwxr-x 1 Unknown+User Unix_Group+1001 9.6k May 30 2004 /cygdrive/p/Recipes/Sacher.doc
-rwxrwxr-x 1 Unknown+User Unix_Group+1001 4.9k May 30 2004 /cygdrive/p/Recipes/Salsa.asc
-rwxrwxr-x 1 Unknown+User Unix_Group+1001 4.9k May 30 2004 /cygdrive/p/Recipes/Salsa.doc
-rwxrwxr-x 1 Unknown+User Unix_Group+1001 2.2k May 30 2004 /cygdrive/p/Recipes/Shrmpstr.asc
-rwxrwxr-x 1 Unknown+User Unix_Group+1001 2.2k May 30 2004 /cygdrive/p/Recipes/Shrmpstr.doc
-rwxrwxr-x 1 Unknown+User Unix_Group+1001 21k Dec 4 2012 /cygdrive/p/Recipes/Spaghetti.odt

$ ls -l //fsfs/ddb/Documents/Recipes/S*
-rwxrwxr-x 1 Unknown+User Unix_Group+1001 9.7k May 30 2004 //fsfs/ddb/Documents/Recipes/Sacher.asc
-rwxrwxr-x 1 Unknown+User Unix_Group+1001 9.6k May 30 2004 //fsfs/ddb/Documents/Recipes/Sacher.doc
-rwxrwxr-x 1 Unknown+User Unix_Group+1001 4.9k May 30 2004 //fsfs/ddb/Documents/Recipes/Salsa.asc
-rwxrwxr-x 1 Unknown+User Unix_Group+1001 4.9k May 30 2004 //fsfs/ddb/Documents/Recipes/Salsa.doc
-rwxrwxr-x 1 Unknown+User Unix_Group+1001 2.2k May 30 2004 //fsfs/ddb/Documents/Recipes/Shrmpstr.asc
-rwxrwxr-x 1 Unknown+User Unix_Group+1001 2.2k May 30 2004 //fsfs/ddb/Documents/Recipes/Shrmpstr.doc
-rwxrwxr-x 1 Unknown+User Unix_Group+1001 21k Dec 4 2012 //fsfs/ddb/Documents/Recipes/Spaghetti.odt

That “Unknown_User” is the signature of this problem, right?

And I can create a file, and read it, but not replace it:

$ echo testing > /cygdrive/p/Recipes/test001.txt

$ ls -l /cygdrive/p/Recipes/test001.txt
----r--r-- 1 Unknown+User Unix_Group+1001 8 Jun 16 13:08 /cygdrive/p/Recipes/test001.txt

$ echo replace the file > /cygdrive/p/Recipes/test001.txt
-bash: /cygdrive/p/Recipes/test001.txt: Permission denied

(note no indication that there is an ACL; which is compatible with the following)

$ getfacl /cygdrive/p/Recipes/test001.txt
# file: /cygdrive/p/Recipes/test001.txt
# owner: Unknown+User
# group: Unix_Group+1001
user::---
group::r--
other:r--

 

My Cygwin setup doesn’t have /etc/passwd or /etc/groups

$ ls /etc/passwd
/usr/bin/ls: cannot access '/etc/passwd': No such file or directory
$ ls /etc/group
/usr/bin/ls: cannot access '/etc/group': No such file or directory
$ ls /etc/groups
/usr/bin/ls: cannot access '/etc/groups': No such file or directory

(Wasn’t absolutely sure of my memory whether group was plural or not 🙁 ; but neither one exists.)

I have configured /etc/nsswitch as I believe is directed (everything left defaults, except change db_gecos to schema “desc”):

$ cat /etc/nsswitch.conf
# /etc/nsswitch.conf
#
# This file is read once by the first process in a Cygwin process tree.
# To pick up changes, restart all Cygwin processes. For a description
# see https://cygwin.com/cygwin-ug-net/ntsec.html#ntsec-mapping-nsswitch
#
# Defaults:
# passwd: files db
# group: files db
# db_enum: cache builtin
# db_home: /home/%U
# db_shell: /bin/bash
# db_gecos: <empty>

db_gecos: desc

$

I have configured the user comment for me, user ddb, in SAM, via the net user command, to have the xml-like Cygwin data in the comment:

$ net user ddb
User name ddb
Full Name David Dyer-Bennet
Comment <cygwin unix="1001" group="Users" />
User's comment
Country/region code 000 (System Default)
Account active Yes
Account expires Never

Password last set 6/2/2017 11:17:13 PM
Password expires Never
Password changeable 6/2/2017 11:17:13 PM
Password required Yes
User may change password Yes

Workstations allowed All
Logon script
User profile
Home directory
Last logon 6/16/2018 12:26:50 PM

Logon hours allowed All

Local Group Memberships *Administrators *fsfsddb
*Users
Global Group memberships *None
The command completed successfully.

And I have also done that for the group:

$ net localgroup Users
Alias name Users
Comment <cygwin unix="1001" />

Members

-------------------------------------------------------------------------------
admin
ddb
localddb
NT AUTHORITY\Authenticated Users
NT AUTHORITY\INTERACTIVE
The command completed successfully.

And I have rebooted the entire Windows box more than once since I last changed anything in the Cygwin config, so the config shown should be in effect when Cygwin produced the output posted above.

(I’ve tried what seems like a million things over the years, but this is the mainline approach to the problem as described in the NTSEC article, and I went through *very* carefully last night to do everything I found in the article about how I was supposed to do it, and documented all the things I’ve done to produce this email).

So…HELP! I feel like this is supposed to be an understood problem, and that I have *done* everything the main article says I’m supposed to do — and it hasn’t helped. Did I miss a step or something? Any ideas? I have this across multiple software versions on both sides and multiple hardware platforms and installations; it’s not some one-time thing.

“Flowers of Vashnoi”

It kind of looks like Lois McMaster Bujold has finished and published the novella she read part of back in 2012 at Dreamhaven, which reading I made a video of. At least, the description sounds very like the story she read.

It sounded very promising, and I’ve just finished the book I was in the middle of, so no doubt I’ll find out for myself soon enough!

The novella on Amazon

The reading on YouTube