Windows freeware unique sort technique for large text files (hosts)

On Sun, 17 Aug 2008 18:46:33 +0200, "B. R. 'BeAr' Ederson"
<br.ederson@expires-2008-08-31.arcornews.de> wrote:

(...)
>
>> The only manual change needed was to move this line back to the top:
>> 127.0.0.1 localhost # this needs to be the first line for some reason
>>
>> I'm digging for the sort command that only sorts from the second line
>> down but haven't found it yet.
>
> The following command line should contain all commands in a one liner:
>
> sed "/127\.0\.0\.1/d" hosts | tr '[A-Z]' '[a-z]' | sort -u | sed
> "1i127.0.0.1 localhost" > hosts
>

Bad idea. My guess of how the OP is using the hosts file is to set
the
IP address of known malicious sites as 127.0.0.1. You'd at least
want
to append your sed's search expression with '[:space:]*localhost'
before
deleting *ALL* lines that contain 127.0.0.1.

sort has a -f option, so the tr is not required.



(...)
> Although the above should work fine, it usually is better to create a
> hosts.new file first and rename it afterwards. But that's up to you.
>

That's actually excellent advice, so much that, that should have been
reflected in your command line. Had the OP used your above command
line,
he'd have lost all entries that corresponded to malicious web-sites
in
his hosts file.


> There are other ways to do the above. I settled with deleting lines
> containing the localhost (127.0.0.1) entries instead of just preserving
> the first line, because the merging of several hosts files may result in
> more than one localhost line...
>

I guess that explains your deleting lines containing 127.0.0.1, but
there
are a number of web sites out there (one even given in an else
thread)
that provide HOSTS files that redirect all requests to known
malicious
sites to 127.0.0.1.


> HTH.
> BeAr

On Sun, 17 Aug 2008 12:07:23 -0700 (PDT), Anand Hariharan wrote:

>> The following command line should contain all commands in a one liner:
>>
>> sed "/127\.0\.0\.1/d" hosts | tr '[A-Z]' '[a-z]' | sort -u | sed "1i127.0.0.1 localhost" > hosts

> Bad idea. My guess of how the OP is using the hosts file is to set the
> IP address of known malicious sites as 127.0.0.1. You'd at least want
> to append your sed's search expression with '[:space:]*localhost' before
> deleting *ALL* lines that contain 127.0.0.1.

You are absolutely right. :-( Actually, it should have been:

sed "/^127\.0\.0\.1/d"...

I thought about adding a filter for possible leading whitespace, since
some hosts files are formatted this way. But the already long command
line got a bit too unreadable. While deleting the whitespace class
operator I must have killed the leading caret on accident... :-(

> sort has a -f option, so the tr is not required.

I used tr, because it already had been suggested in this thread and
will produce nicer looking output. The mixed case result of the sort
process using the -f option is probably harder to look through, if
need arises.

> Had the OP used your above command line, he'd have lost all entries that
> corresponded to malicious web-sites in his hosts file.

At least she would have a more manageable hosts file size. ;-)

BeAr
--
================================================== =========================
= What do you mean with: "Perfection is always an illusion"? =
================================================== =============--(Oops!)===

Donita Luddington <doniludd@sbcglobal.net> writes:

> Do you know if sort can be told to sort all but the first line?

Usually people extract the line, and put it back.
Here is a simple way to do this.

head -1 hosts >hosts.head
sed '1d' hosts | sort <options> >hosts.rest
mv hosts hosts.backup
cat hosts.head hosts.rest >hosts
rm hosts.head hosts.rest

On Aug 17, 10:13*am, Donita Luddington <donil...@sbcglobal.net> wrote:

> What I did was add native Win32 port of the UnixUtils athttp://unxutils.sourceforge.netto my WinXP laptop.
>

If you enjoyed doing this, I STRONGLY recommend instaling Cygwin on
your PC. Since I discovered Cygwin a few years back, it's one of the
first packages that I install on any Windows machine that I need to
work with.
Almost makes me forget that I'm on Windows!