Sort command with very large files

Hi All,

Disclaimer: I'm a newbie.

I am extracting records from many tables of a database into flat files,
and then running a sort command on all the extracted files. The files
are quite big, totalling upto 7 GB for 5 files. My ommand goes like
this -

sort -o final.tmp -k 2,2n -k 3,3 -k 1,1n table1.tmp table2.tmp
table3.tmp table4.tmp table5.tmp

and the output I get is this -

msgcnt 2 vxfs: mesg 001: vx_nospace - /dev/vg02/lvol5 file system full
(1 block extent)
sort: A write error occurred while sorting.

Can anyone please tell me why this problem is occuring? My infra team
assures me that they have 20 GB space in their file system, so 7 GB (or
even double the amount after sorted file is created) would not be
problem. Is it that the sort command has a file size limit? Is it, what
are the workarounds? I don't have bsort in my UNIX. Thanks a lot in
anticipation.

Best,
Attraxion.

2006-12-12, 01:54(-08), attraxion:
> Hi All,
>
> Disclaimer: I'm a newbie.
>
> I am extracting records from many tables of a database into flat files,
> and then running a sort command on all the extracted files. The files
> are quite big, totalling upto 7 GB for 5 files. My ommand goes like
> this -
>
> sort -o final.tmp -k 2,2n -k 3,3 -k 1,1n table1.tmp table2.tmp
> table3.tmp table4.tmp table5.tmp
>
> and the output I get is this -
>
> msgcnt 2 vxfs: mesg 001: vx_nospace - /dev/vg02/lvol5 file system full
> (1 block extent)
> sort: A write error occurred while sorting.
>
> Can anyone please tell me why this problem is occuring? My infra team
> assures me that they have 20 GB space in their file system, so 7 GB (or
> even double the amount after sorted file is created) would not be
> problem. Is it that the sort command has a file size limit? Is it, what
> are the workarounds? I don't have bsort in my UNIX. Thanks a lot in
> anticipation.
[...]

sort uses temp files in /tmp or /var/tmp.

You may try to set the TMPDIR variable to a directory on a
filesystem where there is enough room.

Some sort implementations also have a -T option for that.

--
Stéphane

2006-12-12, 01:54(-08), attraxion:
> Hi All,
>
> Disclaimer: I'm a newbie.
>
> I am extracting records from many tables of a database into flat files,
> and then running a sort command on all the extracted files. The files
> are quite big, totalling upto 7 GB for 5 files. My ommand goes like
> this -
>
> sort -o final.tmp -k 2,2n -k 3,3 -k 1,1n table1.tmp table2.tmp
> table3.tmp table4.tmp table5.tmp
>
> and the output I get is this -
>
> msgcnt 2 vxfs: mesg 001: vx_nospace - /dev/vg02/lvol5 file system full
> (1 block extent)
> sort: A write error occurred while sorting.
>
> Can anyone please tell me why this problem is occuring? My infra team
> assures me that they have 20 GB space in their file system, so 7 GB (or
> even double the amount after sorted file is created) would not be
> problem. Is it that the sort command has a file size limit? Is it, what
> are the workarounds? I don't have bsort in my UNIX. Thanks a lot in
> anticipation.
[...]

sort uses temp files in /tmp or /var/tmp.

You may try to set the TMPDIR variable to a directory on a
filesystem where there is enough room.

Some sort implementations also have a -T option for that.

--
Stéphane

2006-12-12, 01:54(-08), attraxion:
> Hi All,
>
> Disclaimer: I'm a newbie.
>
> I am extracting records from many tables of a database into flat files,
> and then running a sort command on all the extracted files. The files
> are quite big, totalling upto 7 GB for 5 files. My ommand goes like
> this -
>
> sort -o final.tmp -k 2,2n -k 3,3 -k 1,1n table1.tmp table2.tmp
> table3.tmp table4.tmp table5.tmp
>
> and the output I get is this -
>
> msgcnt 2 vxfs: mesg 001: vx_nospace - /dev/vg02/lvol5 file system full
> (1 block extent)
> sort: A write error occurred while sorting.
>
> Can anyone please tell me why this problem is occuring? My infra team
> assures me that they have 20 GB space in their file system, so 7 GB (or
> even double the amount after sorted file is created) would not be
> problem. Is it that the sort command has a file size limit? Is it, what
> are the workarounds? I don't have bsort in my UNIX. Thanks a lot in
> anticipation.
[...]

sort uses temp files in /tmp or /var/tmp.

You may try to set the TMPDIR variable to a directory on a
filesystem where there is enough room.

Some sort implementations also have a -T option for that.

--
Stéphane

2006-12-12, 01:54(-08), attraxion:
> Hi All,
>
> Disclaimer: I'm a newbie.
>
> I am extracting records from many tables of a database into flat files,
> and then running a sort command on all the extracted files. The files
> are quite big, totalling upto 7 GB for 5 files. My ommand goes like
> this -
>
> sort -o final.tmp -k 2,2n -k 3,3 -k 1,1n table1.tmp table2.tmp
> table3.tmp table4.tmp table5.tmp
>
> and the output I get is this -
>
> msgcnt 2 vxfs: mesg 001: vx_nospace - /dev/vg02/lvol5 file system full
> (1 block extent)
> sort: A write error occurred while sorting.
>
> Can anyone please tell me why this problem is occuring? My infra team
> assures me that they have 20 GB space in their file system, so 7 GB (or
> even double the amount after sorted file is created) would not be
> problem. Is it that the sort command has a file size limit? Is it, what
> are the workarounds? I don't have bsort in my UNIX. Thanks a lot in
> anticipation.
[...]

sort uses temp files in /tmp or /var/tmp.

You may try to set the TMPDIR variable to a directory on a
filesystem where there is enough room.

Some sort implementations also have a -T option for that.

--
Stéphane

2006-12-12, 01:54(-08), attraxion:
> Hi All,
>
> Disclaimer: I'm a newbie.
>
> I am extracting records from many tables of a database into flat files,
> and then running a sort command on all the extracted files. The files
> are quite big, totalling upto 7 GB for 5 files. My ommand goes like
> this -
>
> sort -o final.tmp -k 2,2n -k 3,3 -k 1,1n table1.tmp table2.tmp
> table3.tmp table4.tmp table5.tmp
>
> and the output I get is this -
>
> msgcnt 2 vxfs: mesg 001: vx_nospace - /dev/vg02/lvol5 file system full
> (1 block extent)
> sort: A write error occurred while sorting.
>
> Can anyone please tell me why this problem is occuring? My infra team
> assures me that they have 20 GB space in their file system, so 7 GB (or
> even double the amount after sorted file is created) would not be
> problem. Is it that the sort command has a file size limit? Is it, what
> are the workarounds? I don't have bsort in my UNIX. Thanks a lot in
> anticipation.
[...]

sort uses temp files in /tmp or /var/tmp.

You may try to set the TMPDIR variable to a directory on a
filesystem where there is enough room.

Some sort implementations also have a -T option for that.

--
Stéphane

attraxion <attraxion@gmail.com> wrote:
> Hi All,
>
> Disclaimer: I'm a newbie.
>
> I am extracting records from many tables of a database into flat files,
> and then running a sort command on all the extracted files.

Is it possible to avoid the problem by havin the extraction process
produce sorted files in the first place (i.e., SQL "order by")?

--
Warren Block * Rapid City, South Dakota * USA

attraxion <attraxion@gmail.com> wrote:
> Hi All,
>
> Disclaimer: I'm a newbie.
>
> I am extracting records from many tables of a database into flat files,
> and then running a sort command on all the extracted files.

Is it possible to avoid the problem by havin the extraction process
produce sorted files in the first place (i.e., SQL "order by")?

--
Warren Block * Rapid City, South Dakota * USA

attraxion <attraxion@gmail.com> wrote:
> Hi All,
>
> Disclaimer: I'm a newbie.
>
> I am extracting records from many tables of a database into flat files,
> and then running a sort command on all the extracted files.

Is it possible to avoid the problem by havin the extraction process
produce sorted files in the first place (i.e., SQL "order by")?

--
Warren Block * Rapid City, South Dakota * USA

attraxion <attraxion@gmail.com> wrote:
> Hi All,
>
> Disclaimer: I'm a newbie.
>
> I am extracting records from many tables of a database into flat files,
> and then running a sort command on all the extracted files.

Is it possible to avoid the problem by havin the extraction process
produce sorted files in the first place (i.e., SQL "order by")?

--
Warren Block * Rapid City, South Dakota * USA