LINQ to SQL — Can't modify return type of stored procedure

Question

When I drag a particular stored procedure into the VS 2008 dbml designer, it shows up with Return Type set to "none", and it's read only so I can't change it. The designer code shows it as returning an int, and if I change that manually, it just gets undone on the next build.

But with another (nearly identical) stored procedure, I can change the return type just fine (from "Auto Generated Type" to what I want.)

I've run into this problem on two separate machines. Any idea what's going on?

Here's the stored procedure that works:

USE [studio]
GO
/****** Object:  StoredProcedure [dbo].[GetCourseAnnouncements]    Script Date: 05/29/2009 09:44:51 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER OFF
GO
CREATE PROCEDURE [dbo].[GetCourseAnnouncements]
    @course int
AS
SELECT * FROM Announcements WHERE Announcements.course = @course
RETURN

And this one doesn't:

USE [studio]
GO
/****** Object:  StoredProcedure [dbo].[GetCourseAssignments]    Script Date: 05/29/2009 09:45:32 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER OFF
GO
CREATE PROCEDURE [dbo].[GetCourseAssignments]
    @course int
AS
SELECT * FROM Assignments WHERE Assignments.course = @course ORDER BY date_due ASC
RETURN

Answer 1

I've also seen this problem several times and while I don't know what causes it, I've come across a pretty easy way to get past it. It involves manually editing the xml within the .dbml file, but it's a pretty simple edit.

Right-click on your Data Context's .dbml file in the Solution Explorer (not the .layout file nor the designer.cs file) and open it with the XML Editor. You should find your stored procedure listed in a <Function> ... </Function> block. You should also find the custom class you would like to set as the Return Type listed in a <Type> ... </Type> block.

Step one is to give your custom class an identifier. You do so by adding an "Id" tag, like this, making sure that it's unique within the dbml file:

<Type Name="MyCustomClass" Id="ID1">

Step two is to tell your function to use the newly ID'd type as the Return Type. You do so by replacing the line in your <Function> block that looks like

<Return Type="System.Int32" />

with

<ElementType IdRef="ID1" />

Save the file, exit, and rebuild. Done. Re-open the .dbml file in design mode to verify: Your procedure will now have the custom class set as the Return Type.

Answer 2

I had a similar mapping problem, but I found the culprit in my case.

If your procedure or any subprocedure that gets called has temporary objects like

CREATE TABLE #result (
   ID INT,
   Message VARCHAR(50)
)

then you're in trouble, even if you don't select anything of these temporaries.

The mapper has a general problem with these temporary objects, because the type can be changed outside the procedure in the session context. Temporary objetcs are not typesafe for the mapper and he refuses the usage os them.

Replace them by table variables and you're back in business

DECLARE @result AS TABLE (
   ID INT,
   Message VARCHAR(50)
)

Answer 3

I followed the link provided by Tony for a better solution (same answer as Arash's)

• do read the blog, especially the last part, for there is a thing to consider when adding SET FMTONLY OFF in your stored procedure.

When you add

 SET FMTONLY OFF

in the beginning of the stored procedure and load it to DBML,
LINQ2SQL would execute the actual stored procedure.

To get the correct return table object type,
said stored procedure must return something when called w/o parameter(s).
That means:
1. Have default value for all input parameters
2. Make sure SP returns at least a row of data -- this is where I stumbled

create table #test ( text varchar(50) );
insert into #test (text) values ('X'); -- w/o this line, return type would be Int32
select * from #test; -- SP returns something, proper object type would be generated
return;

Answer 4

Okay, I found the problem... kind of. I had changed the name of the table "Assignments" and forgot to update the stored procudure, so the DBML designer was confused. BUT even after I updated the stored procedure, deleted it from the DBML designer and readded it, it wasn't working!

This is nearly the same problem discussed here: http://forums.asp.net/t/1231821.aspx.

It only worked when I deleted the stored procedure from the database and recreated it, and deleted it from the DBML designer, recompiled, restarted Visual Studio, and added it again. This is the second time I've run into "refresh" problems with the Visual Studio DBML designer...

Answer 5

I managed to work out an easier way, which just wasn't obvious at the time, but sounds straight forward when written down:

1. Delete the stored procedure from the design surface of the .dbml file
2. Click Save All files
3. Click Refresh in Server Explorer on the list of Stored Procedures
4. Add (drag) the stored procedure back onto the design surface of the .dbml file
5. Click Save All
6. Click Build
7. Check the designer.cs code file and you will have the updated C# code for the new version of the stored procedure

check http://www.high-flying.co.uk/C-Sharp/linq-to-sql-can-t-update-dbml-file.html

Answer 6

I had the same problem, but only happens if my sp uses FTS, what i did was "cheat" the dbml designer, I remove the fts language stuff and works perfectly, now i can change the return type. Later i go to the sp and add the fts again and works pefectly!. Hope this help.

Answer 7

The way to get around this issue is:

1. Add "set fmtonly off;" to the beginning of your stored procedure.
2. After adding that statement get DBML generate the code for your stored procedure.

If your stored procedure's return type is still 'int' in your DBML code, comment the entire code of stored procedure, create a new SELECT statement whose returning fields types and names match the original's SELECT statement and get DBML regenerate the code again. It has to work!

Answer 8

Thanks @Rubenz, I was also using FTS (Full-Text Search) in a stored procedure and your steps worked.

I commented the FTS section from the stored procedure, added the stored procedure to .dbml, and then uncommented the FTS section back to original.

Answer 9

This also happens when using sql user-defined types as parameters in stored procedures

http://alejandrobog.wordpress.com/2009/09/23/linq-to-sql-%e2%80%94-can%e2%80%99t-modify-return-type-of-stored-procedure/

Answer 10

Better solution found here: http://tonesdotnetblog.wordpress.com/2010/06/02/solution-my-generated-linq-to-sql-stored-procedure-returns-an-int-when-it-should-return-a-table/

Answer 11

OK, I didnt want to be changing anything in my Designer.cs code, I knew there was a different problem and it wasnt related to my stored procedure (I wasnt using temp table anyway).

Simply deleting the sp from the database and updating the model wasnt helping at all. New model created still had the same problems...

What I found is that for some reason a copies of my sp were created in DatabaseModel -> Function Imports.

What I did, I deleted the duplicated objects in Function Imports and updated the model. It worked!

Regards, Chris

Answer 12

I realize that this is an old question but the above suggestions pointed me in the right direction but did not work in my case. I ended up editing the dbml file with the XML editor in Visual Studio as suggested above.

Once in the file, look for the Function section for the stored procedure. You will most likely not see the section – ElementType – which defines the return type. I began to edit the fields from another Function (stored procedure) and found that this was too tedious and may introduce issues.

I decided to delete all the Column definitions from the ElementType - but leave the ElementType section and save the file. I then deleted the stored procedure from the designer and re-added it. It then filled in the correct columns within the ElementType. Worked beautifully.