Out of virtual memory

I have one package that is constantly running out of virtual memory, right now I am on a development server, running only this package. The package is moving data from one table into another on the same server in the same database. The server has 3 Gb of memory and is only running SS2005 and SSIS. I am a local admin on the server and running the package through BIDS, once again for our initial testing. I tried setting the property BufferTempStoragePath to our E drive so it can utilize the 100 Gb of free space we have but that doesn't seem to work either. I have also tried setting the MaxRowSize to many different values to no avail. I am constantly getting an error, see below for exact error, when it gets through roughly half the load. Moreover it reports this error about 500 times in the progress report if I let the package run to completion. Finally, when all is said and done the package has moved the data successfully but the package always shows as failing.

I have googled continuously on this problem but have not found a resolution. I did see on a post here where it was recommended to run the package out of process, however I don't see the benefit at this point when this is the only package I am running. I also don't understand why it would report the error so many times and fail the package when it is completing successfully? Source and Destination have the same number of records at the end of the task. Could someone please try to make sense of this.

Getting Error:
[DTS.Pipeline]

Information: The buffer manager detected that the system was low on

virtual memory, but was unable to swap out any buffers. 4 buffers were

considered and 4 were locked. Either not enough memory is available to

the pipeline because not enough is installed, other processes are using

it, or too many buffers are locked

Thank you for your assistance!write a sql command like "select * from destination_table where 1=0",otherwise SSIS will retrieve all data from your destination table.|||Hi icewill,

Not quite sure I understand why I would do that, I do want all the records. Would you please elaborate? We are loading a data warehouse. Thanks.|||Hello there,

I got such problem a while ago. By reducing "DefaultBufferMaxRow" to certain amount (My package was set to 5000) and keep "DefaultBufferSize" as small as possible (default is 10MB, Max is 100MB), your package should run succefully. Please keep in mind that Test and messure could take little bit of time for reaching the best performance.

Good luck|||Hi

Steglitz,

I have tried numerous settings but none of

them successful. I have tried DefaultBufferMaxRow at the following

values 2000, 5000, 7500, 10000 along with DefaultBufferSize set to the

default, 25Mb, 50Mb and 75Mb.

When running

performance monitor the only difference I noticed was the number of

buffers went up as the values were increased. Here

is an article I found explaining buffer

sizing:

Understanding How the Data

Flow Task Sizes the Buffers

The

data flow engine begins the task of sizing its buffers by calculating

the estimated size of a single row of data. Then it multiplies the

estimated size of a row by the value of

DefaultBufferMaxRows to obtain a preliminary working

value for the buffer size.

If the result is more than

the value of DefaultBufferSize, the engine reduces

the number of rows.

|||Here is another issue to add on top of the previous... Once the package runs Sql Server is left holding all the memory to the system. We watched it both through Performance Monitor and through task manager and after an hour it had not released the memory... 1.4 Gb and holding. The only way we could get it to reduce was to restart the Sql Server service. Is there anyone else out there that seems to be having any of these issues?|||

Bagles,

Actually i dont have any idea about why you need data from destination table, i met the problem that you mentioned before, and at last i found if you select a table as your destination directly or write sql like "select * from destination_table", SSIS will retrieve all data from the database, if there is great number of records in your destination table,that will induce your SSIS out of memory, that is not what i want to see, my mainly purpose is to get the fields to map for, so i modified the sql like "select * from destination_table where 1=0",and the problem has been solved perfectly.

GL~

|||Hi Icewill,

Using a query like that to obtain the metadata makes sense, and I suppose this is probably a silly question, but are you then using an expression to inject the actual sql statement you want at runtime?

I am currently using the DataReader source component which does not allow that, should I be using the OLEDB Source instead? Overall is the DataReader component a bad choice?

Thank you all in advance!|||There are a variety of reasons your server could run out of virtual memory, with both SSIS and SQL2k5 running it, so a couple of questions first.

1. What OS? is it 32 bit or 64 bit?
2. What is the recovery model of your target database?
3. What destination type are you using? SQL Server destination or OLE DB Destination.
4. If you're using an OLE DB destination, what settings are being used, including DataAccessMode, and fast-load settings if applicable?
5. What is the volume of data your moving through the pipeline?

As for icewill's notion that SSIS via SQL is retriving all the data from the destination table, which may be relived by adding "1=0", that assertion is patently untrue and can be observed not to be the case using profiler.

We have partioned tables with billions of rows as target tables and in no case was all the data selected merely by chosing the table as a target, either during design or run-time.|||Greetings Jaegd,

1) We are using Windows Server 2003 Standard Edition with SP1, 32bit.
2) The recovery model is Simple
3) We are using the OLEDB destination with the data access mode of Table Name variable with Fast load.
4) For any package that is running into problems it occurs if there are more than several hundred thousand rows. I currently don't have a definitive threshold but usually if we are moving more than 400,000 rows we run into a problem.

So far here is what we have discovered through several days of testing;

1) It doesn't appear that SSIS is spooling (writing the stream to disk) as it should. The default setting is to the user's temp directory, we have currently set spooling to go to our E: drive due to a lack of space on our C: drive. We have been testing for a solid week and performance monitor has not reported 1 spooled buffer to date. We currently are profiling all of the SqlServer:SSIS Pipeline variables available in Performance monitor as well as the following general categories; SQLServer:Memory Manager, Processor, Physical Disk (all 3 drives), Memory (Available MBytes), and the .NET Data Provider for SqlServer.

2) We finally found the event handler for the logging of BufferResizing and from that we have gathered that SSIS will try to fill the buffer with the MAX amount of memory first (100Mb) and not the default buffer size of 10Mb. Please correct me if I am wrong but that is what we are witnessing.

One thing that would be great to see in the future is the amount of bytes Sql Server is calculating per row and the amount of bytes it is putting in the stream. This would allow people to gain a better understanding of how to use the setting DefaultBufferMaxRows, rather than just guessing at numbers, which is essentially what we have been doing.

3) Finally the biggest adjustment we have made, and which seems to be working, is to limit the amount of memory that Sql Server itself will use. As I stated in a previous post, Sql Server was using all the memory it could get and then not releasing it, which was killing the capabilities of Integration Services. We changed the setting in Sql Server on this particular box to allow Sql Server only half of the available memory which seems to have fixed the problem. Sql Server now will give back the memory that it has used over that setting which frees up enough for SSIS to run properly.

I would still love to hear from you and others that have experienced this problem or have advice on how to avoid it. The above is just what we have witnessed through our own trials and tribulations.

Thank you for your assistance and experience!|||I have taken an alternative route (which may or may not be viable in your implementation). That route is always running SSIS on separate hardware than SQL2k5 (there are licensing issues here naturally). Note, I'm not recommending SSIS/SQL separation for the general case, by the way, since separation adds substantial unnecessary network roundtrips, as well as eliminates the use of the high speed SQL Server destination.

However, SQL/SSIS separation does allow SQL Server "domination" over the memory/CPU on its hardware. Furthermore, and most importantly, separation allows the construction of an SSIS farm, that is, processing over N number of SSIS only machines.

As for x64 vs x86, generally speaking, running x64 SSIS gives me about a 30% boost (from a wall-clock perspective) over running x86 SSIS on the exact same job, an advantage which becomes greater at larger volume levels.

I know this doesn't speak to the spooling issue, or the optimization of the DefaultBufferMaxRows, since virtual memory has become an non-issue since moving to a hardware separation approach.|||Hi jaegd,

I really appreciate your input! Unfortunately I don't have any control over the hardware that is being used; however I do find it beneficial to know how others are using the product and found your use of it a nugget to hold onto for the future! Thanks again!|||

I have been thinking about creating a SSIS farm have you seen any documention on how to set this up and best practises?

I also installed x64 bit version of SSIS, any ideas how I can confirm this?

cheers

Marcus